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ABSTRACT 



The incorporation of published outcome data into clinical 
practice plays a significant role in determining appropriate treatment 
interventions and the efficacy of various modalities. If practitioners are 
willing to conduct their own outcome research, the results will enhance the 
quality of care for clients and improve the quality of information provided 
to funding sources. When simple measures of effectiveness are implemented, 
the demonstrated outcomes from such research can be a very effective tool for 
providing evidence of treatment success. To begin outcome research, 
counselors must have an understanding of efficacy studies and effectiveness 
studies. Counselors can use published efficacy studies to make initial 
choices about treatment interventions, and then conduct effectiveness studies 
on their own practice to measure the success of their treatment. The results 
of effectiveness studies can be useful in helping with allocation of 
resources and in marketing programs to community and health care 
organizations. (Contains 28 references.) (JDM) 
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Abstract 

Mental health practitioners are increasingly being called upon 
to demonstrate the effectiveness of their clinical interventions. 
Effectiveness studies are a type of outcome research that can provide 
useful information to clinicians and to managed care organizations. 

In an age of managed care, counselors are increasingly being called 
upon to demonstrate the effectiveness of their clinical interventions 
(Granello, Granello, & Lee, 1999). The ability to demonstrate treatment 
success is rapidly becoming the standard by which reimbursement is 
judged (Sexton, 1996). In spite of these pressures, many counselors 
have been left unprepared to meet this new standard. Historically, 
mental health practitioners used professional judgment and theoretical 
beliefs to determine treatment interventions. Fee-for-service policies 
and insurance reimbursement were assumed, and insurance companies 
rarely questioned treatment decisions (Plante, Couchman, & Diaz, 
1995). In the current practice environment, however, counselors who 
cannot demonstrate their successes may find themselves unable to 
survive professionally (Burlingame, Lambert, & Reisinger, 1995). 

Although the demonstration of treatment effectiveness is 
increasing in importance, many mental health professionals and 
agencies have resisted participation in outcome measures, and there is 
widespread resistance among mental health professionals to beginning 
their own assessment programs (Plante, et al. 1995). Studies have 
revealed that the vast majority of mental health practitioners report 
that they do not read research or engage in research and believe that 



research has little or no impact on their counseling practices (Cohen, 
Sargent, & Sechrest, 1986; Falvey, 1989). In 1983, Norcross and 
Prochaska found that when presented with 14 reasons to select a 
particular approach or orientation with a client, the psychologists in 
their study rated outcome research 10th, just above “family 
experiences” and “own therapist’s orientation.” More recently, Norcross 
(2000) noted there was little evidence that this ranking had improved 
significantly during the past 17 years, although he predicted that the 
recent emphasis on the importance of outcome research should result 
in increased reliance on such research in the future. A recent survey 
found that although the majority of the clinical diplomates of the 
American Board of Professional Psychology (65%) supported the 
development of empirically supported treatments, the majority of 
respondents (54%) did not routinely use them in their practices (Plante, 
Anderson, & Boccaccini, 1999). 

Both philosophical and practical concerns have been identified at 
the root of the resistance to engaging in outcome research and 
incorporating research results into practice. Philosophically, some 
providers have argued that the invasion of accountability into mental 
health care has negatively affected therapeutic decision making 
(Sherman, 1992). Some argue that the therapeutic process itself is not 
quantifiable (Mirin & Namerow, 1991) or that clinical flexibility, 
clinical judgment, and creative expression of theory should be valued 
more than scientific method and statistical analysis (Havens, 1994). 
Still others argue that time spent in evaluation could be better used in 
treatment (Plante, et al. 1995). Even among clinicians who are willing 
to conduct outcome research, practical concerns often stand in the way. 
Practitioners may erroneously believe that the task will be 
overwhelming or that a program of research will necessarily be costly, 
complex, and time-consuming (Granello et al., 1999). What has become 
apparent is that few mental health practitioners have received the 
training they need to conduct such research. Research methods courses 
in university programs often focus on understanding laboratory research 
with true experimental designs that are often impossible to implement 
in real-world assessment (Sandell, Blomberg, & Lazar, 1997). Thus, 
practitioners may be ill prepared to conduct their own outcome research, 
regardless of their willingness to do so. 

The incorporation of already published outcome data into clinical 
practice plays a significant role in determining appropriate treatment 
interventions and the efficacy of various modalities (Sexton, 2000). 
Bridging the gap between research and practice is essential (Whiston 
& Coker, 2000). However, if a practitioner is willing to conduct his or 
her own outcome research, in conjunction with already published 
research to support general clinical interventions, the result will be 
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enhanced quality of care for clients and improved quality of information 
provided to funding sources (Granello, Granello, & Lee, in press). 
Measuring treatment effectiveness need not be a difficult or 
cumbersome task. Simple measures of effectiveness can be 
implemented quite easily, and the demonstrated outcomes from such 
research can be a very effective tool for providing evidence of treatment 
success. 



Methodological Considerations 



To engage in outcome research, counselors must first have an 
understanding of the two main types of research that are used to 
demonstrate clinical success: efficacy studies and effectiveness studies. 
Efficacy studies use random assignment to treatment and control group, 
manualize treatment, and use participants who meet criteria for a single 
diagnosed disorder (Seligman, 

1995; Wampold, 1997). Additionally, there are clearly defined 
inclusion and exclusion criteria for clients and an adequate sample 
size to obtain the necessary statistical power (Fishman, 2000). Efficacy 
studies provide useful information and are appropriate designs for 
laboratory studies or settings in which highly controlled manipulation 
of variables is possible (Sandell et al., 1997). However, these studies 
are very expensive and time-consuming and often are funded through 
a university or through a grant offered by a foundation or a 
pharmaceutical company. 

Effectiveness studies, on the other hand, attempt to answer how 
well clients fare under treatment as it is actually practiced in the field. 
Such studies yield useful and credible information that can empirically 
validate psychotherapy (Lambert, Huefner, & Nace, 1997). 
Effectiveness studies recognize that less-than-methodologically-ideal 
situations exist in the field. Among these situations are that (a) therapy 
is not always of fixed duration, and typically continues until the client 
improves or quits or until insurance coverage runs out; (b) 
psychotherapy often is eclectic rather than manualized and typically is 
self-correcting (e.g., if one technique is not working, then another 
usually is tried); (c) clients typically present with multiple problems, 
some subclinical and some diagnosable, rather than the pure diagnoses 
represented in efficacy studies; and (d) psychotherapy in the field 
typically is concerned with improvements in general functioning rather 
than in specific symptom relief, which is the typical measure in efficacy 
studies (Seligman, 1995). 

Efficacy and effectiveness studies have different strengths and 
limitations. Efficacy research typically has high internal validity but 
low external validity. The conditions under which efficacy research is 
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conducted are so structured that there is a high degree of confidence 
that changes that occur are due to the treatment, not to confounding 
variables. However, the conditions under which efficacy research is 
conducted are often so dissimilar to what happens in the field that 
there is a low degree of confidence in generalizing the results of a 
particular study to field conditions. Conversely, effectiveness studies 
have high external validity but low internal validity. Because they 
sample a population directly from the field, there is a high level of 
confidence that results can be generalized to other members of the 
population (Fishman, 2000). The lack of a control group and of therapist 
adherence to specific treatment interventions are noteworthy, however, 
and lead to concerns about confounding variables (e.g., the passage of 
time) that might affect treatment results (Granello et al., 1999). Overall, 
efficacy and effectiveness studies provide complementary research 
designs. Counselors can use published efficacy studies to make initial 
choices about treatment interventions, then conduct effectiveness 
studies on their own practice to measure the success of their treatment 
(Granello & Hill, 2000). 

Research Design 

Research design is guided by the research questions under 
investigation (Granello & Hill, 2000). What specific information does 
the counselor wish to have about his or her practice or clients? Clinicians 
wishing to engage in tracking the success of a single client for 
reimbursement purposes would ask different research questions than 
would those wishing to investigate their treatment success with their 
overall client load or with clients having particular disorders (e.g., 
anxiety disorders). 

Many effectiveness studies follow a pre-post or pre-post-follow- 
up design. That is, clients are given an instrument or series of 
instruments upon entering treatment, and the same instrument or 
instrument battery is given at discharge, and if desired, at pre-designated 
follow-up periods (typically 3, 6, or 12 months, or all three). Other 
types of effectiveness studies track the progress of a single client at 
various points in treatment (e.g., every week, every month), on a specific 
rating scale, with results that can be represented graphically to 
demonstrate progress. Still other studies use existing data from client 
records (e.g., Global Assessment of Functioning scores) to make 
comparisons over time or across client groups. Thus, for a single client, 
the counselor may choose to measure the reduction of a very specific 
symptom and engage in a single-case pre-post design, using a repeated 
measures r-test, or may choose to forego statistical analysis in favor of 
a graphic representation of multiple data points. To measure symptom 
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data, our recommendation is to keep the data collection and 
analysis manageable. 

2. Although outcome research need not be cost prohibitive, some 
foresight will be necessary to set aside sufficient funds for 
instruments and, if necessary, data analysis. We have found 
that university-agency collaboration, although not necessary, 
can provide a symbiotic relationship (data for the university, 
data analysis for the agency). 

3. As much as possible, the collection of data should be integrated 
into clinical practice (e.g., put pretests in admissions packets 
so they are not forgotten). 

4. For clinicians not currently collecting data, any step, however 
small, is a step in the right direction. Collecting data on 
treatment effectiveness can provide both an external benefit 
in terms of marketing and an internal benefit in validating and 
improving clinical success. 
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